United States Patent Application 

FOR 



Four-Color Mosaic Pattern For 
Depth and Image Capture 



Inventor: 



TlNKU ACHARYA 



Prepared by: 



Blakely, Sokoloff, Taylor & Zafman, 
12400 Wilshire Boulevard 

Seventh Floor 
LOS ANGELES, CA 90025-1026 

(503) 684-6200 



Express mail No. fv 37.5528247 US 



F OUR-CQLQR MOSA ICPAIIERNSQE 
nPPTH AND IMAGE CAPTURE 

This U.S. Patent application is a continuation-in-part of U.S. Patent application 
numberl0/376,156,medFebraary28,2003,entitled"FOUR-COLORMOSAIC 

PATTERN FOR DEPTH AND IMAGE CAPTURE." 

PF.T A TED APPT ICATIONS 

This U.S. Patent application is related to U.S. Patent application number 
10 /376,127, filed February 28, 2003, entitled "SUB-SAMPLED INFRARED SENSOR 
FOR USE IN A DIGITAL IMAGE CAPTURE DEVICE." 

TPPHNTr.AL FIELD 

[0001] Themventionrelatestothefieldofimagecapture. More particularly, the 
inventions toasensor for capture of an image and depth information and uses 

thereof. 

BACKGROUND 

[00021 Digital cameras and other image eapture deviees operate by capturtng 
electromagnetic radiation and measuring the intensity of tire radiation. The spectral 
content of electromagnetic radiation focnsed onto a focal p.ane depends on, among other 
things, .he image ,0 be captured, the illumination of the subject, the transmission 
characteristics of the propagation path between tire image subject and tire optical system, 

the materials used in tire optical system, as weH as tire geometric shape and stze of the 

optical system. 
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,0003] For consumer ima^ng systems (e.g., digital cameras) me spectral range of 
intere s, is .he visible regron of .he e.ectromagnetic spectrum. A common method for 
preventing difficulties caused hy radiation outside of .he visual range is .0 use ionieal.y 
colored glass or a thin-film optical coating on glass .0 create an optical element .ha. 
passes visihle Ugh. (typical* having wavelengths in me range of 380 nm ,0 7S0 rnn). 
This clemen. can he p!aced in front of .he taking lens, within tire lens system, or i. can be 
incorporated into the hnager package. The principal disadvantage ,0 mis approach is 
increased system cost and complexity. 

[0 „„4, A color filter array (CFA) is an array of flhers deposited over a pixe! sensor 
array so .hat each pixel sensor is substantially sensitive .0 on!y tire e.ectromagnetic 
radiation passed hy tine finer. A filter in tire CFA can be a composite filter manufactured 
from multiple filters so tira, tire .ransfer fimction of tire resulting filter is tire product of 
the transfer functions of the constituent filters. Each filter in the CFA passes 
etecttomagnetic radiation wtthin a particular spectra, range (e.g., wavelengths that are 
interpreted as red). For exanrp.e, a CFA may be composed of red, green and blue filters 
so that the pixel sensors provide signals indicative of tire visible color spectrrun. 
100051 If mere is no. an infrared blocking element somewhere in the optical chain 
infrared OR, radiation (typical* considered .0 be Ugh. with a wave.ength greater than 
780 nm) may a.so be focused on the focal plane, hnaging sensors or devices based on 
srlicon technology typically require .be use of infiared blocking elements to prevent >R 
from entering tire tnraging array. Silicon-based devices will .ypical.y be sensitive ,0 light 
with wavelengths up ,0 ,200-, !f tine IR is permitted, o enter tire array, tire device will 
respond to the IR and generate an image signal including tire K. 
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,0006! In current ttaee-dtntensiona, cameras, the depth information is captured 
blu e (visib.e color) images at fixed time — Pu.ses of IR Ught are —ed 

m pulse are collected between the capture of the visible co.ors. 
,0007, The number of bits available to the ana.og-.o-dig.ta. converter detennines the 
d epth increments that can be measured. By applying accurate timing to cut off imager 
integration, the infiared light can directly carry shape information. By controlling the 
nation operation after pulsing the IR fig*, ft. camera can detemtine what interval of 
ais.anee will measure object depth and such a technic can provide the shape of fte 
ejects in the scene being captured. This depth generation process is expensive and 
beavi.y dependent on non-silicon, mainly optica, and mechanica, systems for accurate 
shutter and timing control. 
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ppttjf r>F.SCRIP Tl"N ™ THF nB A WINGS 

The invention is iUustta.ed by way of example, and notby way of limitation, in 
to figures of me aeeompanying drawings in whieh like referenee numerals refer to 
similar elements. 

Figure 1 is an example Bayer pattern that ean be used to eapture eolor image 

data. 

Figure 2 illustrates one embodiment of a sub-sampling pattern that ean be used to 
capture color and depth information. 

Figure 3 is a block diagram of one embodiment of an image capture device. 

Figure 4 is a flow diagram of one embodiment of an image capture operation that 
.ncludes interpolation of multiple color intensity vahtes including infrared intensity 
values. 

Figure 5 illustia.es one embodiment of a sub-sampling pattern that can be used to 
capture color and depth information. 

Fig »re 6 il.ustia.es one embodiment of a sub-sampling pattern tha. ean be used .o 
capture color and depth information. 

Figure 7 illus.ra.es one embodiment of a sub-sampling pattern that ean be used to 
capture color and depth information. 

Figure 8 illustrates one embodiment of a sub-sampling pattern that can be used to 
capture color and depth information. 
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DETAILED DE SCRIPTION 

[0008] In the following description, for purposes of explanation, numerous specific 
details are set forth in order to provide a thorough understanding of the invention. It will 
be apparent, however, to one skilled in the art that the invention can be practiced without 
these specific details. In other instances, structures and devices are shown in block 
diagram form in order to avoid obscuring the invention. 

[0009] A sensor for color and depth information capture is disclosed. A filter passes 
selected wavelengths according to a predetermined pattern to the sensor. The sensor 
measures light intensities passed by the filter. In one embodiment, the wavelengths 
passed by the filter correspond to red, green, blue and infrared light. The intensity values 
can be used for interpolation operations to provide intensity values for areas not captured 
by the sensor. For example, in an area corresponding to a pixel for which an intensity of 
red light is captured, interpolation operations using neighboring intensity values can be 
used to provide an estimation of blue, green and infrared intensities. Red, green and blue 
intensity values, whether captured or interpolated, are used to provide visible color image 
information. Infrared intensity values, whether captured or interpolated, are used to 
provide depth and/or surface texture information. 

[0010] A color image pixel consists of three basic color components - red, green and 
blue. High-end digital cameras capture these colors with three independent and parallel 
sensors each capturing a color plane for the image being captured. However, lower-cost 
image capture devices use sub-sampled color components so that each pixel has only one 
eolor component captured and the two other missing color components are interpolated 
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based on the color information from the neighboring pixe.s. One pattern common* used 
for sob-sampled color image captnre is the Bayer pattern. 

,0011] Flgurel is an example Bayer pattern that can be nsed to captnre color image 
data, m the description herein sensors are described as captining color intensity vahres 
for individnal pixe!s. The areas for which color intensity is detemtined can be of any size 



or shape 



,0012] Each pixel in the Bayer pattern consists of only one color component - cither 
red (R), green (G) or bine (B). The missing components are reconstructed based on the 
values of the neighboring P ixe, values. For example, the pixel a. location (3,2) contains 
only blue intensity infomration and the red attd green components have been filtered out. 
,0013) The missing red information can be obtained by interpolation. For example, 
the red intensity information can be oblamed by determining the average intensity of the 
four adjacent red pixels at locations (2,1), (2,3), (4,1) and (4,3). Similarly, the missing 
^een intensity information can be obtained by determining the average intensity of the 
four adjacent green pixe.s a, locations (2,2), (3,1), (3,3) and (4,2). Other, more complex 
insolation techniques can a.so be used. However, an image capture device using the 
standard Bayer pattern cannot capture depth information without additional components, 
which increases the cost and complexity of the device. 

,0014] Figure 2 illustrates one embodiment of a sub-sampling pattern that can be 
used to capture color and depth information. Use of a four-color (R, G, B, IR) mosaic 
pattern can be used to captiue color information and depth information using a single 
sensor. As described in greater detail below, missing color intensity information can be 
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the four colors are captured contemporaneously. 

, 00151 For example, me pixe. in location (7,3) corresponds to blue intensity 
information (row 7 and column 3). Thus, i, is necessary .0 recover green and red 
ta ^ t y i nforma n onmomermpmvideamUco,orpix. 1 .Recoveryofmm te n S i W 

valu esofthefournei g hho ri n gg reenpixe„ocations(7, 2 ),(7,),(6,3,and(S,3)isusedfor 
t he^nin t en S i t yva.ueofpixe.(7,3).Sim«arl y ,meavera g eof,he 1 n,eos, r yva,uesof 

<n\ \ n i\ (<s Tl and (9 3) is used for the red 
the nearest neighbor red pixel locanons (7,1), (7,5), (5,3) and CM) 

intensity value of pixel (7,3). The B intensity information for pixel (7,3) can be 
(8,2) and (8,4). 

l00 16, One embodhnen. of a technique for interpolating coior and)or depth 

convention. Alternate techniques can also be used. 

For the pixel X(m,n) in location (tn.n) = 

case 1= (both m and n are odd integers) 

IR = — ~ 4 

G = 4 
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if X(m,n) is RED, then 

R = X{m,n) ; 

X( m -2,n) + Xim + 2, n) + X(m, n - 2), X(m, n + 2) . 
B = 4~~ 

else 

B = X(m,n) ; 

X(m -2,n) + X(m + 2, n) + X{m, n - 2), X(m, n + 2) 
R- 4~ 

end if 

case 2: (m is odd and n is even) 

X(m-l,n) + X(m + l,n) 
/* = 2 ' 

G = X(m,n) ; 

if X(m,n-1) is RED, then 
i? = ^(w,n-l) ; 
5 = ^(m,« + l) ; 

else 

5 = ^(w,«-l) ; 
# = JT(m,n + l) ; 
end if 

case 3: (m is even and n is odd) 

X(m,n-l) + X(m,n + \) 
IR- " ' 

G = X(m,n) ; 

if X(m-l,n) is RED, then 
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R = X(m-\,n) ; 
B = X(m + \,n) ; 

else 

B = X(m-l,n) ; 
R = X(m + \,n) ; 

end if 

case 4: (both m and n are even integers) 

IR = X(m,n) ; 

X(m -U)^(m^^ ; 
G = 4 

if X(m-l,n+l) is RED, then 



else 



X(m- 


\,n + \) + X(m + \,n-\) . 

J C — / 


X(m- 


2 

.\.n-l) + X(m+\,n + l) 




: ■ / 

2 


X{m- 


-hn + l) + X{m + \,n-\) 

7 • " j 


X{m 


2 

-\.n-\) + X(m + \,n+\) 



end if 

end 

[00171 Figure 3 is a block diagram of one embodiment of an image capture device. 
Una system 3.0 foenses light from a scene on sensor nni. 320. Any type of lens system 
known in the at, for raking images can be used. Sensor unit 320 includes one or more 
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sensors and one or more filters sueh that the image is captured using the pattern of Figure 
2 or similar pattern. In one embodiment, sensor unit 320 includes a complementary 
metal-oxide semiconductor (CMOS) sensor and a color filter array. Sensor unit 320 
captures pixel color information in the pattern described above. Color intensity 
information from sensor unit 320 can be output from sensor unit 320 and sent to 
interpolation unit 330 in any manner known in the art. 

[0 018] Interpolation unit 330 is coupled with sensor unit 320 to interpolate the pixel 
color information from the sensor unit. In one embodiment, interpolation unit 330 
operates using the equations set forth above. In alternate embodiments, other 
interpolation equations can also be used. Interpolation of the pixel data can be performed 
in series or in parallel. The collected and interpolated pixel data are stored in the 
appropriate buffers coupled with interpolation unit 330. 

[0019] In one embodiment, interpolation unit 330 is implemented as hardwired 
circuitry to perform the interpolation operations described herein. In an alternate 
embodiment, interpolation unit 330 is a general purpose processor or microcontroller that 
executes instructions that cause interpolation unit 330 to perform the interpolation 
operations described herein. The interpolation instructions can be stored in a storage 
medium in, or coupled with, image capture device 300, for example, storage medium 
360. As another alternative, interpolation unit 330 can perform the interpolation 
operations as a combination of hardware and software. 

[0020] Infrared pixel data is stored in JR buffer 342, blue pixel data is stored in B 
buffer 344, red pixel data is stored in R buffer 346 and green pixel data is stored in G 
buffer 348. The buffers are coupled with signal processing unit 350, which performs 
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signal processing functions on the pixel data from the buffers. Any type of signal 
processing known in the art can be performed on the pixel data. 
[0021] The red, green and blue color pixel data are used to generate color images of 
the scene captured. The infrared pixel data are used to generate depth and/or texture 
information. Thus, using the four types of pixel data (R-G-B-IR), an image capture 
device can capture a three-dimensional image. 

[0022] In one embodiment, the processed pixel data are stored on storage medium 
360. Alternatively, the processed pixel data can be displayed by a display device (not 
shown in Figure 3), transmitted by a wired or wireless connection via an appropriate 
interface (not shown in Figure 3), or otherwise used. 

[0023] Figure 4 is a flow diagram of one embodiment of an image capture operation 
that includes interpolation of multiple light intensity values including infrared intensity 
values. The process of Figure 4 can be performed by any device that can be used to 
capture an image in digital format, for example, a digital camera, a digital video camera, 
or any other device having digital image capture capabilities. 
[0024] Color intensity values are received by the interpolation unit, 410. In one 
embodiment, light from an image to be captured is passed through a lens to a sensor. The 
sensor can be, for example, a complementary metal-oxide semiconductor (CMOS) sensor 
a charge-coupled device (CCD), etc. The intensity of the light passed to the sensor is 
captured in multiple locations on the sensor. In one embodiment, light intensity is 
captured for each pixel of a digital image corresponding to the image captured. 
[0025] In one embodiment, each pixel captures the intensity of light corresponding to 
a single wavelength range (e.g., red light, blue light, green light, infrared light). The 
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nS-o.hep.enocaUonsfonowsap,— edpa«em. Onepa,*. 
and 8 Additional alternate patterns ean also be used. 

,,„*, The eaptnred co!or intensrty values from rhe sensor unit are sent ,0 an 

tensity interpolauon operations on the captured intensity values, 420. Or one 
de.rioedwiflrrespecttotnee.uationsanove.Formepanemsof^resS^.TandS 

c ,mnl P with a different color intensity pattern, other 
alternate embodiments, for example, with a ome 

interpolation equations can be used. 

l0M „ The sensor unit captures intensity values for vistb.e co.ors as we,, as for 
interpolated color intensity values. 

, 00M1 The infrared intensity values are also interpolated as described herein. The 
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values can be selectively interpolated. 

100291 ^ecap-co.o.in.ensi.yv^^a.ein^cCo.^values 

part o ft Keca P »eaev i ceo rth e m e m o ry ca„ b ee Xt en 1 aHo,o rrem o l e & o m , tl >ecap,u re 
^.■noneeo^ao^uffersa^.ed.os.orerea.^btueand^ 

used. 

,„„,„, An ou,pu, image is generated u S ,ng, for exampie, a signai processing 

.e^e^eo^i^ge.^ou.pu.irr.geeanbedisp.aMs.ored.p^e.c. 
|00311 Flg „re 5 DM- one embodimen, of a sub-sampling partem *a« can be 

" <R - G ' B ' m)m ° SaiC 

panen.eanbe^.oeap^eo.o^onna.ionanddep^onna.onusingaaing.e 
^Asde.crtbed.ngrea.erd^be.ow.missingeoioHn^—ioneanbe 

the four colors are captured contemporaneously. 
[00 32, Onecmbodin^ 

information follows. Alternate techniques can also be used. 

For the pixel X(m.n) • in location (m,n) : 

easel: (both m and n are odd integers) 
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2 



R= 2 
G = — 4 



# = X(m,«); 
case 2 : (m is odd and n is even) 

jk = — ~ 4 



R = X(m,n) ; 



G = 



B= 2 



case 3: (m is even and n is odd) 
IR = X{m,n); 

£ = — 4 



case 



G = 2~ 

X{m-l 1 n)^ 
B- J* 

4s (both m and n are even integers) 
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G = X{m,n); 

5 = — " 4 

end 

,0033, Big-re 6 illustrates one embodiment of a sub-sampling pattern ,ha, can be 
^.oc^eco.o.andoep.h—on.Useofafonr.eo.o^R.G.B.^mosaie 

pauenr ean be used to capture co.or imfonnation and depth —ion using a smgie 
^ As described in greater detad beiow, missing color intensity information can be 

the four colors are captured contemporaneously. 

l00341 Oneembodimentofateehniqueforinte^atingcoloraud/ordepm 
information follows. Alternate techniques can also be used. 

For the pixel X(m,n) in location (m.nl = 

easel: (both tn and n are odd integers) 

IR = X(m,n) ; 

X(m-hn) + X(m + \,n) m 
G = J 

if X(m,n-1) is RED, then 
R = X(m,n-\) ; 
B = X{m,n + \) ; 

else 
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B = X(m,n-\) ; 
R = X(m,n + \) ; 
end if 

case 2: (m is odd and n is even) 

y fm-l.nH^(itt + l") + ^( w 2n zlhjfe^ ; 
IR = ■ 4 

G = 4 

if X(m,n) is RED, then 
R = X{m,n) ; 

B = 4 

else 

B = X(m,n) ; 

X(m-2 r,) + X(m + Zn)+X(m,n-2) + X{ n h n+^ , 
i?= 4~ 

end if 

case 3: (m is even and n is odd) 

x{m ^^Y(r^^X(m-\,nVx{m + \,n) . 
ZR = - 4"" 

G = X(m,n) ; 

if X(m-l,n-l) is RED, then 

X(m-\,n-\)+X(m + hn +]) 
R = — ^ ' 
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else 

X(m-hn-\)+X(m + l,n + \) . 



B = 



R= 2 
case 4: (both m and n are even integers) 
IR = X(m,n) ; 



X(m,n-\) + X(m,n + l) 



G = 2 
if X(m-l,n) is RED, then 
R = X{m-l,n); 
B = X(m + l,n) ; 

else 

B = X(m-\,n); 
R = X(m + \,n) ; 
end if 

end 

[00351 Figure 7 illustrates cue embodiment of a sub-sampling pattern that can be 
used ,o eapture coior and depth information. Use of a four-eolor (R, G, B, HQ mosaic 
pattern can be used to capture color information and depth information using a single 
sensor. As described in greater detail below, missing cote intensity information can be 
intend using neighboring intensity value, In one embodiment, tntensity values for 
the four colors are captured contemporaneously. 
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[0036] One embodiment of a technique for interpolating color and/or depth 
information follows. Alternate techniques can also be used. 

For the pixel X(m,n) in location (m,n) : 

case 1: (both m and n are odd integers) 



IR = 

X{m,n-\)+X{m,n + l) 
R = ~ ' 



X(m-\,n) + X(m + l,n) 
G= 2 



B = X(m,n); 

case 2 : (m is odd and n is even) 

X(m-\,n) + X{m + \,n) 
IR = — ; ' 



R = X(m,n) ; 

y(g ^n-l)+ Y( m + \.n-\) + X(m-l,n + \)+X(m+ } 1 n^ , 



G = 



X(w,n-l)+X(m,« + l) 

case 3: (m is even and n is odd) 

X(m,n-l) + X(m,n + \) 
IR = 2 "' 



G = ; 
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case 4: (both m and n are even integers) 
IR = X(m,n) ; 



X(m-ln) + X(m + \,n) . 

IT 



B = 



end 

100371 Figure 8 illustta.es one embodimeu. of a sub-sampliug pattern .hat ean be 
used «o capture eolor and deptb informal. Use of a four-color (R, G, B, fR) mosaic 
pattern ean be used to eaplure eolor information and depth information using a single 
sensor. As described in greater detail be.ow, missing eolor intensity information ean be 
interposed using neighboring intensity values. In one embodiment, intensity values for 
the four colors are captured contemporaneously. 

10038] One embodiment of a technique for interpolating color and/or depth 
information follows. Alternate techniques can also be used. 

For the pixel X(m,n) in location (m,n) : 

case 1: (both m and n are odd integers) 

X{m-\,n) + X(m + \,n) 
IR = 2 

Xim-ln-^X^n^ ; 
R = 4 
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„ X(m,n-\) + X(m,n + l) 
G = - ~ 



B = X{m,n); 
case 2: (m is odd and n is even) 

IR = 4 

X(m-\,n)+X(m + \,n) 



X(m -l.»-l) + X(m + l,«-l) + ^-l." + i)±^±li^ ; 



L 

G = X(m,n) ; 

B X(m,n-\)+X{m,n + \) _ 

case 3: (m is even and n is odd) 
IR = X(m,n); 

R = X(m-l,n) ; 

Y( m ^„-ti+X(m + ln-\)+X(m-h n + i)+X(m + l,n + \) . 
G = I - 



X{m-ln)+X(m + \,n) 
B = ' 2 



case 



4: (both m and n are even integers) 

rn X(m,n-l)+X{m,n + \) . 
/* = j 

i? = X(»7,»); 



^On-l^ + XCm + l.n) 
G= - 



-l.»-n + X(m + l,» + l) + X(m + l,»-l)+^feLz2^ll : 



end 
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[0039] Reference in the specification to "one embodiment" or "an embodiment- 
means that a particular feature, structure, or characteristic described in connection with 
the embodiment is included in at least one embodiment of the invention. The 
appearances of the phrase "in one embodiment" in various places in the specification are 
not necessarily all referring to the same embodiment. 

[0040] In the foregoing specification, the invention has been described with reference 
to specific embodiments thereof. It will, however, be evident that various modifications 
and changes can be made thereto without departing from the broader spirit and scope of 
the invention. The specification and drawings are, accordingly, to be regarded in an 
illustrative rather than a restrictive sense. 
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